From 69.6% to a Defensible E&M Coding Engine: Separating the LLM's Reading from the Rules That Decide

Challenge

A clinical-documentation AI company focused on orthopedics shipped a production feature that selects E&M codes from the note, but had no rigorous read on how often it was right, where it failed, or what the feature was worth to a practice. E&M selection is a place where being approximately correct is not enough: the code has to survive payer review and an audit, which means the logic behind it has to be explainable and reproducible, not the opaque output of a single model call. The company needed to measure production accuracy against clinician ground truth, mine claims and payer notes for denial and fee-schedule intelligence, and put a defensible dollar figure on the product, all in a way that would hold up to scrutiny.

Approach

The core method was to confine the LLM to what it is genuinely good at (reading a clinical note and extracting facts and language) and to move every decision that determines the code into deterministic, unit-tested code. The model reads; versioned rules and tables decide. That separation is what makes the output auditable, because each coding decision can be traced back to a specific rule rather than to a probabilistic guess.

• Stood up a full local replication of the production AI coding environment with byte-identical payload reconstruction, so prompts and models could be A/B tested under exactly the conditions production runs under, against a 713-case clinician-validated ground-truth set.
• Built a decoupled structured-MDM architecture that separates LLM fact-extraction from a deterministic AMA Table-1 rule engine, implementing the 2-of-3 rule, Category 1/2/3 data counting, and risk triggers as explicit, unit-tested rules rather than leaving them to the model.
• Built an ICD-10-to-SNOMED crosswalk that constrains the AI to clinically valid codes, so the model cannot select a code outside the allowed set.
• Built an LLM denial-note-mining pipeline over thousands of payer notes, organized around a dual-axis root-cause taxonomy, to turn free-text denial language into structured root causes.
• Used the ground-truth harness as the validation backbone: every prompt or model change was scored against clinician labels before it could be considered an improvement.

Impact

• Established a production accuracy baseline of 69.6% on the 713-case ground-truth set, giving the company a measured starting point it did not have before.
• Lifted weighted accuracy to about 95% on the 713-case clinician ground-truth set, up from the 69.6% baseline, by moving the coding decision out of the model and into the deterministic, unit-tested rule engine. Accuracy reflects the tested configuration and is not a guarantee for any single practice.
• The deterministic structured-MDM rule engine mechanically fixed 29% of the baseline rule-application errors, a class of mistakes that the rules close out by construction rather than probabilistically.
• The SNOMED crosswalk won 83.8% of cases head-to-head and cut the invalid-code rate down from a 47.5% baseline, materially reducing clinically invalid selections.
• The LLM note-mining pipeline classified 100% of payer notes versus 68% for the keyword-only approach, at about $6.62 per 1,000 notes, so coverage improved without a meaningful cost ceiling.
• Produced a data-backed value estimate of about $2,283 per 1,000 encounters, drawn from a multi-practice ROI study of 6,509 encounters, which translates to an estimated $1.4M to $1.7M annualized for a large practice. This is a modeled estimate built from the study data, not realized revenue, and it scales with encounter volume and the practice’s payer mix.

Capabilities demonstrated

• Byte-identical local replication of a production AI environment for controlled A/B testing of prompts and models.
• Clinician ground-truth construction and a scoring harness that gates every change on measured accuracy.
• Hybrid architecture design that isolates LLM extraction from a deterministic, versioned, unit-tested rule engine (AMA Table-1, 2-of-3, data counting, risk).
• Terminology engineering (ICD-10-to-SNOMED) to constrain model output to clinically valid codes.
• LLM-driven denial-note mining with a structured root-cause taxonomy over thousands of payer notes.
• ROI modeling that ties accuracy gains to a defensible per-encounter dollar value.

Anonymized by design: client names stay off the narrative per our reference policy. Figures that are modeled, small-sample, or targets are identified as such above.