Validating AI and machine learning models under CBUAE MMS — the explainability obligation

The credit decision that takes thirty seconds to produce is, from the borrower’s perspective, a seamless experience. From the risk and regulatory perspective, it is an output of a model — typically a machine learning model trained on historical credit outcomes, transactional data, and alternative data sources — that must meet the same governance standards as any other model used to drive material financial decisions.

The speed and scale at which ML-based credit scoring operates does not exempt it from the CBUAE Model Management Standards. The standards apply to any model used in a material decision-making context, regardless of methodology. This includes gradient-boosted trees, neural networks, ensemble scoring systems, and the large class of vendor-provided models marketed under the label of AI or alternative data scoring. The institution that deploys one of these models is subject to the same requirements for model documentation, independent validation, and ongoing monitoring as the institution that built its own logistic regression scorecard in-house.

What differs is the difficulty of meeting those requirements with ML models. The validation challenges are genuine, and the institutions that handle them well tend to have invested in understanding what makes ML validation different before designing the validation programme.

What the CBUAE MMS requires

The CBUAE Model Management Standards address independent validation through Article 10, and third-party model governance through Article 4.7.

The four validation components in Article 10 — conceptual soundness, data quality and processing, performance testing, and outcomes analysis — apply to ML models directly, but each requires adaptation.

Conceptual soundness for a traditional logistic regression means assessing whether the model’s theoretical basis is appropriate for the application, whether the variable selection is justified, and whether the coefficients behave as expected. For an ML model, conceptual soundness assessment must address whether the training objective is aligned with the credit decision objective, whether the feature set captures the intended signals without embedding illegitimate proxies (such as features that correlate with protected characteristics), and whether the model’s architecture is appropriate for the data structure and decision context.

Data quality and processing for a traditional model means checking the training and validation data for completeness, accuracy, and representativeness. For an ML model, it additionally means checking the training data for historical biases — periods of unusual economic conditions, demographic shifts, product-mix changes — that could produce a model that performs well in backtesting and poorly in deployment. The feature engineering pipeline, which is often more complex in ML models than in traditional scorecards, requires its own documentation and quality assessment.

Performance testing for a traditional model means discriminatory power (Gini, KS statistic), calibration (predicted PD versus observed default rate), and stability (PSI). These apply to ML models as well, but ML models introduce the additional dimension of interpretability testing — the ability to explain, at the observation level, what features drove the prediction and in which direction. This is not a traditional validation metric; it is a requirement that emerges from the regulatory expectation of explainability.

Outcomes analysis for both model types means comparing model-driven decisions against realised outcomes. For ML models, this requires particular attention to cohort analysis — whether the model performs consistently across demographic and product cohorts, or whether its discrimination degrades in sub-populations that were underrepresented in the training data.

The explainability gap

The specific challenge of ML model validation that has no direct equivalent in traditional model validation is the explainability gap. Traditional scorecards are, by construction, interpretable. The scorecard tells you the points assigned to each attribute band, and the credit officer can trace any score to its component inputs. ML models — gradient boosted trees, neural networks, random forests — are generally not interpretable in this sense. The output is a score; the pathway from inputs to score is not directly readable from the model structure.

A model that a credit officer cannot explain to a regulatory inspector is a model risk exposure, not just a technical gap.

The explainability gap is not merely a technical inconvenience. In the UAE, the CBUAE’s consumer protection framework requires that institutions be able to explain adverse credit decisions to borrowers. The broader governance expectation — from the CBUAE Corporate Governance Standards, from the MMS use test, and from supervisory dialogue — is that the institution understands its own decision-making tools. A model that produces scores the institution cannot explain is a tool the institution does not fully control.

The practical response to the explainability gap is the use of post-hoc interpretability methods — SHAP (SHapley Additive exPlanations) values, LIME (Local Interpretable Model-Agnostic Explanations), and equivalent tools that decompose individual predictions into feature contributions. These methods provide approximate interpretability rather than inherent interpretability, but they can produce a reasonably tractable answer to the question of which features drove which prediction, and in which direction.

For validation purposes, post-hoc interpretability methods should produce outputs that the validator can use to assess whether the model’s behaviour is consistent with its stated purpose — whether it is, in fact, using the intended credit signals rather than proxies or artefacts. Where the interpretability analysis reveals that a significant portion of the model’s discrimination comes from features that do not have an obvious credit-theoretic justification, further investigation is required before the validation can be completed.

Vendor models and third-party governance

A material share of ML-based credit scoring in the UAE is provided by third-party vendors — fintechs, data providers, and international credit scoring platforms. Article 4.7 of the CBUAE MMS requires that third-party models be subjected to the same validation standards as internal models. The obligation sits with the institution, not the vendor.

The vendor opacity problem — where the algorithm is a commercial black box and the vendor will not disclose the feature weights or model structure — is not unique to the GCC. It is a challenge that regulators across major jurisdictions are addressing progressively. The current CBUAE expectation, as reflected in the MMS and in supervisory dialogue, is that the institution must be able to assess model performance, demonstrate it on local data, and produce evidence of ongoing monitoring — even if the internal mechanics of the model are not fully disclosed. Where that standard cannot be met, the model’s deployment requires explicit risk acceptance at the board level, not simply an acknowledgment that the vendor is reputable.

Feature drift and ongoing monitoring

ML models are particularly susceptible to feature drift — the gradual change in the statistical properties of input features that occurs as the economic environment, the customer population, and the product mix evolve. A model trained on pre-pandemic data is not the same model operating in the post-pandemic environment, even if nothing in the model itself has changed. The training relationships between features and credit outcomes have shifted; the model’s discrimination may have degraded without the degradation being immediately visible in headline performance metrics.

The CBUAE MMS requires periodic revalidation and ongoing performance monitoring for all models. For ML models, this monitoring needs to include stability analysis on input features — Population Stability Index (PSI) at the feature level, not only at the score level — and early warning metrics for discrimination degradation by cohort. Where a model covers a customer population that has changed significantly since training (post-crisis shifts in employment patterns, new product segments, regulatory-driven portfolio composition changes), a targeted revalidation of the affected segments is warranted before the full periodic revalidation cycle.

The institutions that have invested in ML model governance ahead of supervisory scrutiny tend to be those that have adopted a continuous monitoring posture rather than a periodic validation posture. The difference is not in the quality of the periodic validation itself, but in the institution’s ability to detect degradation in between validation cycles and to respond before the degradation produces material credit losses or supervisory observations.

For a deeper treatment of ML model validation methodology — including SHAP-based interpretability assessment, feature drift monitoring, and the documentation patterns that satisfy CBUAE MMS Article 10 in supervisory dialogue — see our white paper in the Library.