The first two pieces of this series toured the file drawer of model governance and the human chain around it. This one is the pocket edition: five questions that fit into a single meeting with a fund, a platform, or anyone selling an AI-flavoured investment process. No question requires a technical answer from you — the reaction is the data.
Question one: what exactly does the model decide?
Ask them to finish a short, concrete template: “The model decides ___.” Models occupy very different positions in the workflow: some merely rank lists a human then trades on; some recommend position sizes; some fire orders without a person in the loop; some do nothing but score clients for marketing. Governance effort should match the position — and so should your attention.
- A good answer sounds like: “The model generates target weight bands each night; a portfolio manager must approve any change outside the bands before execution.” Concrete, bounded, and quietly implies both a human gate and a working definition of abnormal.
- A bad answer sounds like: “Our proprietary AI optimises your portfolio for the future.” Untestable poetry. The larger the claim and the vaguer the decision description, the further the firm is from knowing where its own machine acts.
Question two: where are the validation dates?
The single most expensive fact to fake in model marketing is when. Ask when the current version was last independently tested, on what out-of-sample window, and when the next review falls due. Strong programmes can recite this; brokers who never asked the question cannot.
- A good answer names a reviewer who is not the builder, a date within the current model generation, a test window the model was not tuned on, and a re-validation calendar (annual for most material models, sooner after major changes).
- A bad answer offers “rigorously backtested” — which is the pedigree of every failed quantitative fund in history — or a validation so old photographs of the review session would need restoration. The piece on reading the paperwork lists what a real validation contains; the date is where honesty is easiest to check.
Question three: who can switch it off, and where is that logged?
Kill-switch access is the fastest audit of an oversight chain. If nobody at the table can answer immediately, the honest summary of the governance regime is: whatever the marketing says, no stop exists.
- A good answer names the role on duty overnight, describes the logged intervention history and mentions at least one event where a human overrode the machine — and what the firm changed afterwards. Counter-intuitively, firms that describe their interventions proudly tend to be the safe ones.
- A bad answer suggests the question has never been rehearsed: hesitation, committee deflection, or reassurance that the model “has never needed” halting. The 2012 Knight Capital incident, covered in the oversight piece, is the industry’s standing proof that the ability to stop is worth more than decades of uninterrupted operation.
Question four: what did the model do in the last stress event?
Performance lawns get mowed and presented; stress behaviour has to be asked for. Every serious allocator wants the specific account of one named shock — March 2020 is the modern standard, with its circuit breakers and spreads — told from the model’s side: what it signalled, what it did, what humans intervened to do.
- A good answer mixes the outcomes and the interventions honestly. “Down with the market, drawdown within mandate, and our monitor tripped two de-risking thresholds — here is how many hours the intervention took.” Ask the follow-up question a footprint leaves: what was changed in monitoring or limits afterwards, because that’s the culture speaking.
- A bad answer pivots to a backtest of the crisis or a smooth “the model handled it well” with no numbers, no timing, and no interventions — an answer implying either no oversight happened or nobody thought you’d ask.
Question five: who profits when the model is wrong?
The uncomfortable close of the series: incentives are observable even when code is not. Someone is paid on volume, on spread, on assets gathered, monthly regardless of the model’s actual behaviour — and every one of those structures answers this question for you, silently.
- A good answer is a fee table shown without hesitation: flat and transparent fees, penalties aligned with the signals, and no reward stream that grows while client outcomes shrink. Where conflicts survive commercially, they are disclosed in the grainy detail the “conflict of interest” label usually flattens.
- A bad answer describes the fee structure of the firm and not the model: a revenue line that fattens on activity the model itself encourages — churning, product-linked recommendations, data edges never disclosed. Watch for the phrase “aligned incentives” offered flat, with no arithmetic behind it.
How to run the meeting
Sequencing matters less than note-taking discipline, but two practices sharpen the whole exercise. First, ask for dates attached to nouns: a validation review, a version, an intervention, a stress test, a fee change — every governance noun in a healthy system carries a date, and answers that float free of the calendar are prose, not evidence. Second, separate the roles in the answers: the person briefing you may honestly not know an item, and “I will confirm with our model risk team” is an acceptable answer once; twice for the same link, three meetings in a row, is a diagnosis.
A compact way to record the results is a simple two-column page — question left, and on the right the four possible outcomes: concrete (date, name, number), partial (concrete but with gaps acknowledged), deflected (reassurance vocabulary), or absent (nobody knows). Two concretes and three acknowledged gaps describe a firm in progress. Five deflects describe a brochure with an org chart attached.
One caution against over-reading: small firms and young products legitimately run thinner files than global institutions — the questions are not a indictment of scale, they are a test of self-knowledge. A two-person team that answers “here is what we do not yet have, here is when we will have it” is passing the test. A large institution whose only concrete answer is its founding year is failing it.
The residue: watching how the answer arrives
Run the five questions and you will notice something beyond the content: the latency. Teams with genuine governance answer halting questions in one breath — they have answered them for auditors, boards and their own internal reviews. Teams without it need emails, callbacks and a polished follow-up. That gap is the most reliable instrument you own, and it costs exactly nothing to use.
The final residue belongs to you, not the firm: the five questions only work if you are willing to hear an answer that changes nothing. If no answer could move you, governance questionnaires become theatre on your side of the table too — and the models keep running either way, unilluminated and unwatched. Asking is free; evasion answers more than numbers ever do.