Most codification captures what an expert can write down — steps, rules, checklists. But transcripts and memos record what was done and what was said about it, never the tradeoff that actually decided it. This lays out an approach for recovering that implicit reasoning — the criteria, their conditional weights, and the constraints — so an agent can reason in new situations instead of imitating old ones.
The dominant way to make an agent act like a specific expert is to feed it that expert's trail: chat transcripts, meeting recordings, written memos, a curated memory of past cases. A language model summarizes the trail into rules, procedures, and examples. The agent then follows them.
This works until the situation is one the trail never covered. Then the agent either freezes, applies the nearest rule out of scope, or invents a rule that sounds right. The failure is structural, not a matter of more data: the trail records outputs of reasoning and a partial narrative of it. The reasoning itself — a judgment over competing criteria under constraints — was never written down, because the expert never had to write it down, and often could not.
Three findings from outside AI say this is not a fringe case. Polanyi: experts know more than they can tell. Argyris and Schön: what practitioners say governs their decisions and what actually governs them diverge reliably. Nisbett and Wilson: people confidently explain choices whose real causes they had no access to. A stated rationale is evidence, not ground truth.
The rest of this paper works through a single venture-capital decision to show where the hidden reasoning lives, then lays out a process — with a statistical model at its center and a language model doing the semantic lifting — for recovering it.
A general partner at a $200M early-stage fund has capacity for one more Series A lead this quarter. Three opportunities are on the table. The partner picks one, writes the memo, and the investment committee approves. Everything below is fictional but representative.
This is the material a codification pipeline would ingest: the investment memo and the partner meeting transcript. Both are honest. Both are incomplete.
We recommend leading Deal B's Series A. The founder's prior execution as CTO of a portfolio company gives us unusual conviction in the team. The market timing for inference infrastructure is favorable and the entry price is attractive relative to comparable rounds. Deal A is a strong company but the round is competitive and priced accordingly. Deal C shows excellent growth but carries regulatory exposure we are not positioned to underwrite.
Read the transcript again. Four things are doing the real work, and none of them appear in the memo as criteria. Two are barely spoken at all.
| What actually drove the decision | In the memo | In the transcript | Operating |
|---|---|---|---|
| Ownership floor ≥ 15%, must lead | |||
| No unlicensed money movement (this fund) | |||
| Fund-narrative weight, high in year 4 | |||
| Known founder substitutes for diligence | |||
| Growth, price, team quality (the stated criteria) |
Next quarter, Deal D arrives: AI infrastructure, founder is a known ex-CTO from the network, strong market narrative — and a competitive round at $80M post, where the fund could get 9% as a follower.
A behavior twin trained on the memo and transcript has learned "AI infra + known founder + market timing → invest." It recommends Deal D. The partner would pass without a second thought: the ownership floor is a constraint, and it is not in the artifact because it was never at risk in Deal B and therefore never discussed as a criterion.
A decision has three layers. Today's pipelines read the top two and produce an artifact that looks like the third.
The memo is a projection of L3 onto a document genre. It is written to persuade a committee, so it foregrounds criteria that sound like analysis (market timing) and suppresses ones that sound like bias (I know this founder) or institutional need (we need a logo for the next raise). This is not dishonesty. It is what memos are for.
The transcript is closer, but still only surfaces criteria that were contested in the room. The ownership floor was mentioned because Deal A threatened it. Had all three deals offered 20%, it would never have come up — and the codified artifact would have no idea it existed.
Four kinds of thing are systematically missing, and each is missing for a different reason:
| Hidden element | Deal B example | Why the record misses it |
|---|---|---|
| Hard constraints | Ownership ≥ 15%; lead position | Constraints are only spoken when threatened. Most decisions never threaten them. |
| Conditional weights | Fund-narrative weight depends on fund year | The condition (year 4) is ambient context the speaker doesn't restate. The record has the weight at one point, never the function. |
| Tradeoff ordering | Narrative > price > growth, this quarter | Orderings are revealed by choices, not stated. One decision reveals one comparison. |
| Substitutions | Known founder substitutes for technical diligence | Expressed as a feeling ("I know how he ships"), not as the procedural exception it actually is. |
This is the case for a different target. Inverse reinforcement learning made the same move two decades ago: don't imitate the demonstrator's policy, recover the reward function that makes the policy optimal, then re-derive the policy anywhere. A digital twin that copies behavior is apprenticeship without the reward. A reasoning twin recovers the reward — criteria, conditional weights, constraints — and derives the behavior. It transfers to Deal D. And it is arguable: a partner can look at "ownership floor 15%" and say "actually it's 12% for AI infra," which they cannot do with a policy.
Two tiers. Plain language explains what the process does and who does what. Full detail gives the model, the fitting procedure, and the math. Same content, different altitude.
The process has three roles, and the design principle is that each does only what it is good at.
The language model reads the record and proposes what the criteria might be, what alternatives were on the table, and what question to ask next. It is good at meaning and bad at weighing. The statistical model takes the proposed criteria and the observed choices and estimates how much each criterion mattered, under which conditions. It is good at weighing and cannot invent a criterion. The human is never asked to state a weight — only to make a judgment on a concrete case. Experts are reliable at the second and unreliable at the first.
Turn the record into a list of decisions, each with the situation, the choice, and — critically — the alternatives that were rejected. The transcript shows what was done, not what was passed on. The LLM reconstructs the alternatives from the record and, where the record is silent, proposes plausible ones for the human to confirm.
Deal B: situation = {fund year 4, three deals, one slot}; chosen = B; alternatives = {A, C, pass on all}. Plus the prior six years of this partner's deals, each reconstructed the same way.
Read the memos and transcripts and propose what the partner might be weighing — generously. Over-proposing is fine; the fit in step 3 will zero out what does not matter. Also propose situation features that might change the weights: fund year, sector heat, whether the partner has a personal relationship with the founder.
Proposed criteria: growth, price, ownership, team quality, founder relationship, market narrative, regulatory exposure, competitiveness of round. Proposed context: fund year, dry powder remaining, recent portfolio loss.
Given the instances and the criteria, estimate how strongly each criterion pulled the choice, and whether that strength depends on the situation. The model is deliberately simple and interpretable — an expert produces dozens of decisions, not millions — and it outputs a small table: this criterion, this weight, under these conditions.
Output: ownership has a sharp cliff near 15% (looks like a constraint, not a preference). Market-narrative weight is near zero in fund years 1–2, dominant in years 4–5. Regulatory exposure flips from mild negative to near-veto after the loss in year 2.
Look for pairs of decisions that look the same on the current features but went opposite ways. These are conflicts. Instead of averaging them away, ask the LLM: what could distinguish these two cases? Add its proposal as a feature and refit. A conflict is a missing variable until proven otherwise.
Two AI deals in year 1 with strong founders — both passed. Deal B in year 4 — invested. The features cannot separate them. LLM proposes "years remaining in deployment period." Refit: separation is clean.
Pick the single question whose answer would most reduce uncertainty in the weights — usually a minimal variation of a real case, at a suspected boundary — and ask the expert to judge it. Never "what are your criteria?" Always "would you still, if this one thing were different?"
"You led Deal B at 20%. Same deal, same founder, but $50M post and you get 10% as a follower. Still in?" — "No." One answer converts a suspected constraint into a confirmed one. "What if 14%?" — "…probably. If we lead." Threshold found.
For each decision region, measure how predictable the choice is once the criteria are known. Highly predictable and stable → codify as a rule. Unpredictable even after context discovery → codify as guided judgment with the weights and contrasting examples attached. Irreversible downside → constraint, regardless. Section 06 covers this; section 05 makes it track the evidence over time.
The agent carries the criteria explicitly and scores new situations against them. Every time it escalates (situation outside the evidence) or is overridden (human disagrees), that is a new instance. Return to step 3.
Deal D arrives. Ownership at 9% violates the constraint. The agent passes and can say exactly why. No memo ever said "ownership floor."
A decision instance is a situation s, a chosen action a, and the set of alternatives A(s) that were available. Criteria are functions g_i(s, a) ∈ ℝ scoring how well action a serves criterion i in situation s. Hard constraints are indicator functions h_j(s, a) ∈ {0, 1}. The decision function is modeled as a conditional choice model:
This is a Bradley–Terry / Luce choice model — the same form as the reward model in RLHF — with two deliberate changes. First, weights are conditional: w(s) is piecewise constant over a small set of regions R_k defined by scope conditions. The regions are the codified context; the weights are the codified tradeoff. Second, the probability is meaningful: P ≈ 1 means the tradeoff is settled in that region; P ≈ 0.5 means genuine indifference or a variable not yet found. Lexicographic priority is the limit w_i / w_j → ∞. A hard constraint is an infinite penalty, and is represented separately rather than as a large weight so that it is never traded.
ssituation — the VC case: {fund year, deals on the table, dry powder, recent loss, …}
a, A(s)the choice and the alternatives — {invest A, invest B, invest C, pass}
g_i(s,a)criterion score — growth multiple, ownership %, team score, narrative fit, regulatory exposure
R_kregions — e.g. {fund year ≤ 2}, {fund year ≥ 4}, {post-loss}
w_i^(k)weight of criterion i in region k — the thing to recover
h_jconstraints — ownership ≥ 15% ∧ lead; no unlicensed money movement (post-loss region only)
Behavioral cloning fits π(a | s) directly. It has no notion of why and no way to score an action it has not seen taken in a situation like s. Inverse RL fits the reward R such that the observed π is (near-)optimal, and derives π* = argmax R anywhere. The model above is a linear-in-criteria, region-conditional reward with the argmax softened into a choice probability. The softening matters: it lets the fitted P carry information about how settled each tradeoff is, which section 06 uses directly.
The choice of a low-capacity, interpretable model is not a limitation to apologize for. An expert produces tens to low hundreds of instances. Anything richer than a sparse logistic model over LLM-proposed features will overfit, and — more importantly — cannot be argued with by the expert whose reasoning it claims to represent.
Instances. From transcripts, memos, and logs, extract (s, a, A(s)). The alternatives are load-bearing: the model learns from differences g(s,a) − g(s,a'), so a missing or invented alternative directly corrupts the fitted reward. Where the record does not state alternatives, the LLM proposes them and they are queued for human confirmation in 4.5 before they carry weight.
Criteria. The LLM proposes candidate g_i from narrative (L2) and from action patterns (L1) independently, and candidate region features from ambient context. Over-propose. The L1 penalty in 4.4 prunes. Criteria that appear in both L1- and L2-derived sets start with higher prior confidence; criteria in only one set are flagged as "stated but not revealed" or "revealed but not stated" — the latter are the interesting ones.
For a fixed region assignment, Bradley–Terry reduces to logistic regression on feature differences. For each instance and each rejected alternative a' ∈ A(s) \ {a}, construct a row Δ = g(s, a) − g(s, a') with label 1. Fit:
Regions are found by fitting a shallow decision tree (depth ≤ 2) over the candidate context features with the residuals of a single-region fit as the target; each leaf becomes an R_k. Constrain regions to be few and nameable — a region the expert cannot recognize as a real situation is a fitting artifact.
Constraints are detected rather than fit. A criterion whose fitted weight is very large and whose observed instances never fall below a threshold τ on the chosen action — while alternatives below τ were rejected — is promoted to a candidate constraint h: g_i(s,a) ≥ τ and confirmed by elicitation (4.6). In the VC case, ownership behaves this way: no chosen deal below 15%, several rejected deals below it, and a fitted weight that the L1 penalty keeps trying to shrink but cannot.
After fitting, compute the per-instance log-likelihood. Instances with low likelihood are those the model "disagrees with." Cluster them by φ(s). A cluster of poorly-explained instances whose situations are near-identical on current features but whose choices diverge is a conflict, and there are only three explanations:
Formally, for a conflict cluster C, the LLM proposes features z_1 … z_m; each is scored by the information gain on the choice labels within C; the top feature is added if its held-out improvement exceeds a threshold. In the VC case, the year-1 AI passes and the year-4 AI investment form exactly such a cluster, and "years remaining in deployment period" has the highest information gain of the proposals.
The expert's time is the scarce resource, so each question must be chosen to maximize expected information gain over the posterior on w. With a Laplace approximation to the posterior, this is standard Bayesian active learning for logistic regression: choose the query pair (s, a, a') that maximizes
In practice the highest-EIG queries are minimal variations of real cases near a suspected boundary: take an actual instance, perturb one feature toward the region where the model is uncertain, and ask "still?" The expert answers a concrete pairwise judgment — never a weight — and the answer is a new row in the fit. Constraint candidates from 4.4 are confirmed the same way: perturb the suspected constraint below τ and ask; a flip confirms the constraint and a bisection locates τ.
Alternatives proposed by the LLM in 4.3 are confirmed here too: "Was passing on all three genuinely on the table?" A rejected alternative that was never real is removed before it distorts the fit.
| Element | Region | Value | Provenance |
|---|---|---|---|
| h_own: ownership ≥ 0.15 ∧ lead | all | constraint (τ from bisection: 0.14–0.15) | 17 instances; 2 elicitation answers |
| h_reg: no unlicensed money movement | post-loss (year ≥ 3) | constraint | 4 instances; 1 elicitation; transcript "not this fund" |
| w_narrative | year ≤ 2 | 0.08 ± 0.11 | 9 instances |
| w_narrative | year ≥ 4 | 0.91 ± 0.14 | 6 instances; 1 elicitation |
| w_founder_rel | year ≥ 4 | 0.62 ± 0.20 | 6 instances — substitutes for diligence step |
| w_growth | all | 0.78 / 0.44 by region | 23 instances |
Illustrative values. The ± is the posterior standard deviation; wide intervals are where the next elicitation query should go. Every row traces to the instances and answers that produced it, so any row can be disputed by the expert and re-fit without it.
Section 04 recovers the decision function from one batch of evidence. But the partner's weights drift — a fund ages, a loss leaves a scar, a market heats up — and each round of codification is itself noisy: a few new deals, a handful of answers. Two mistakes are available: overwrite what you knew with a noisy new fit, or ignore real change because it disagrees with the old artifact. A state-space filter is the principled middle.
Think of the partner's real decision function as a hidden state — the weights and thresholds that are actually in force this quarter. You never see it directly. What you see, each period, is a reading of it: the weights the fit in section 04 produces from that period's deals and elicitation answers, together with how uncertain that fit is (the ± column in the fitted table).
The update rule is the one every navigator and every Kalman filter uses. Carry forward what you believed last period, allowing for some drift. Take the new reading. Then move your belief toward the reading by an amount that depends on which is more trustworthy: if the reading is noisy and your prior is sharp, barely move — shrink toward the prior. If the prior is vague and the reading is precise, move most of the way — adapt to the observation. The amount you move is the gain, and it is computed, not chosen.
The gain reduces to a two-by-two: how sure were we before, and how clean is the new reading?
Let x_t be the hidden decision function in period t: the stacked region weights w^(k) and constraint thresholds τ. Let y_t be the estimate of those quantities produced by the section 04 fit on period-t instances and elicitation answers, and R_t its Laplace posterior covariance — the ± column in the fitted table. Then:
The random-walk transition is deliberate: we have no model of how a partner's weights evolve, only that they do, slowly. Q_t is the drift allowance. The scalar case makes the shrinkage reading explicit: K = P/(P+R), so the update is x̂_t = (1−K)·prior + K·reading — a precision-weighted average, identical in form to James–Stein / empirical-Bayes shrinkage with the prior as the shrinkage target.
Rt is not estimated separately. It is the posterior covariance of the period-t Bradley–Terry fit (4.4), which already reflects how many instances the period contained, how well-separated they were, and whether elicitation answers were present. A period with two deals and no questions produces a wide R_t and is shrunk away; a period with a bisection on the ownership floor produces a near-degenerate R_t on τ_own and is adopted.
Qt is the one genuinely free quantity, and it should be small by default and adaptive on evidence. Use the normalized innovation:
A single large innovation is an anomaly and is absorbed by the gain — that is the point of shrinkage. A run of large innovations is a regime change, and the correct response is to stop trusting the prior on the affected components. Inflating Q does exactly that; it is the temporal analogue of adding a region feature. Once the LLM names the regime ("post-loss", "year ≥ 4"), it becomes a region in the section 04 model and Q returns to baseline — the state has been re-indexed, not merely widened.
Thresholds τ_j live in x_t with two modifications. Their transition noise is near zero — constraints do not drift, they get re-decided — and their observation noise from instances alone is set to infinity: no number of chosen deals above 15% and rejected deals below it can move τ_own on its own, because that pattern is equally consistent with any threshold in the gap. Only elicitation (a bisection answer, R → 0) updates a threshold, and when it does, K ≈ 1. The filter thus reproduces the rule-like semantics of a constraint without special-casing it.
For a firm with several partners, initialize each partner's state at the firm-level mean μ with prior covariance Σ_between, estimated from the dispersion of partner-level fits. A new partner or a partner with few deals is then shrunk toward the firm; a partner with a long record earns their own weights. The between-partner dispersion is also diagnostic: a criterion whose weight varies widely across partners and does not separate by region is the multi-expert values conflict of 4.5, and is surfaced for the committee rather than averaged.
Section 06 places each region on the spectrum using entropy of the choice probability. With the filter in place, two more quantities are available per region and per weight: the state uncertainty P_t and the drift allowance Q_t. The placement rule becomes:
| State condition | Reading | Placement |
|---|---|---|
| P small, Q small, H ≈ 0 | Stable, settled, predictable | Rule or procedure |
| P small, Q small, H high | Genuinely a judgment; we know it well | Guided judgment with weights attached |
| P large | Under-evidenced | Guided judgment; queue elicitation query |
| Q inflated (regime flagged) | Something changed and is unnamed | Escalate; re-elicit; hold prior artifact as provisional |
| τ with R → 0 from elicitation | Confirmed constraint | Constraint |
The consequence is that the codified artifact — the rules, the guidance, the constraints, the escalation list — is a function of the filtered state, and is regenerated each period. Placement drifts with the evidence; nobody re-decides it in a meeting.
The last question in any codification is "should this be a rule the agent follows, or a judgment the agent makes?" With a fitted choice model, this stops being a design meeting and becomes a measurement.
For each region R_k — each named situation the model found — ask five questions in order. The first that applies wins. Every input to these questions is a quantity the fitted, filtered model already produces.
| Question | If yes | Why |
|---|---|---|
| Is the downside irreversible or high-blast-radius? | Constraint on the downside, regardless of what else the region does | A wrong judgment call that can be undone is a learning event. One that cannot is a loss. Cost asymmetry beats predictability. |
| Is the choice near-deterministic (low entropy) and stable across refits? | Procedure or scored heuristic | The expert always does this here. Judgment adds variance without adding value, and rules are cheap to run at volume. |
| Is the entropy high but the region reducible — a variable might still be found? | Guided judgment: emit weights, the contrasting examples, and the open question | The agent should decide, but with the tradeoff made explicit and the evidence attached, so its choice can be audited and its exceptions harvested. |
| Has the filter flagged a regime change on this region? | Escalate and re-elicit; hold the prior artifact as provisional | Persistent surprise (section 05) means a variable changed that the model does not name yet. Acting on the old rule is acting on a stale state. |
| Is the situation outside the provenance set? | Escalate — at runtime, dynamically | Any region can be pushed to escalation for a specific case. A reasoning twin that knows what it does not know is worth more than one that guesses. |
Two failure modes bracket this. Over-codification produces an agent that applies a rule outside its provenance and cannot tell it has done so. Under-codification produces an agent that re-litigates settled questions inconsistently. The spectrum is the mechanism for placing each region between them deliberately — and, because placement is a function of the filtered state, re-placing it automatically each period as the evidence changes. Nobody re-decides "is this a rule?" in a meeting; the state does.
The argument in one line: an agent built from transcripts inherits a policy; an agent built from a recovered reward inherits a way of deciding. The first is a behavior twin and fails at Deal D. The second is a reasoning twin and passes on Deal D for the right reason.
The single model — a conditional choice function with explicit constraints — unifies four things that today are handled separately and by hand: tradeoff weights (fitted), context (regions), conflict (residuals that point at missing variables), and the rule-versus-judgment decision (entropy). Wrapping it in a state-space filter adds the fifth: time. Each round of codification becomes a noisy reading of a drifting hidden state; the gain decides how much to believe it; persistent surprise flags a regime change and sends the LLM to name it. Provenance comes free, which means every element of the codified artifact can be disputed by the expert it claims to represent — and every element carries a date and an uncertainty.
w, R_k, and h explicitly in its context and score against them, rather than reconstruct a rationale afterward — and that the same recovery process should be run on the agent's decisions, periodically, to check that it is still the twin it was deployed as.Q — would let the twin anticipate change instead of only tracking it. And with quarterly cadence and a handful of deals per period, the innovation test has few degrees of freedom; how early a regime change can be called with honest confidence is open.The practical shift is small in mechanics and large in posture. Stop asking experts to write down their reasoning. Ask them to make judgments on cases the model chose, let the model do the weighing, and let the language model do the reading. The reasoning was never in the transcript. It was in the choices, waiting to be asked the right question.