Checklists at the Speed of Prefill: Local Clinical Eligibility Screening with Open Decision Models
Preprint status: This paper has been submitted to arXiv and is still in moderation. This post is a preview. We’ll add the arXiv link when it goes live.
Code, app, data generators and every prediction: github.com/menonpg/clinical-system1
The question
A lot of standard-of-care work is applying a published checklist to a free-text chart. Does this patient qualify for lung-cancer screening? For an AAA ultrasound? For this clinical trial?
On this blog we’ve covered the new wave of open decision models: Kev, Laya and Cloudflare’s Clef. They don’t generate text. They answer typed yes/no questions with calibrated probabilities from a single prefill pass, which makes them a natural fit for checklists. This paper tests whether they work for this task on one laptop, and which practical steps make them safe to use.
The setup

- One yes/no question per criterion. We encoded the USPSTF 2021 lung-cancer and 2019 AAA screening recommendations this way.
- A conjunctive score. P(eligible) is the product of the required answers. Every “no” names the criterion that failed, so each decision can be audited.
- Ten local systems. Nine decision models from four families (Kev 0.8B–27B, Laya, H2O-Lightning-4B, Clef-flash 9B), plus Gemma 4 as a generative baseline. Everything ran on a single M1 Max.
- Three test sets:
- 900 synthetic notes from held-out templates with exact per-criterion labels
- 40 hand-written notes
- 600 eligible-versus-excluded patient–trial pairs from TREC Clinical Trials 2021
What we found
1. Newer decision models are much better. On lung-screening template notes, AUROC was 0.977 for Kev-27B and 0.976 for Clef-flash 9B, compared with 0.877 for Kev-4B. Zero-shot Laya was near chance.
2. Arithmetic is the weak spot. Most errors came from criteria that need a calculation, such as pack-years or years since quitting. The fix is simple: a deterministic computed-facts hook works out those numbers and adds them to the note. With the hook, AUROC on hand-written notes rose to 0.997–1.000 for the strongest models. Kev-27B could then auto-decide 95% of hand-written notes at ≤5% error.
3. A small model fine-tuned on a laptop can catch up. An 85-minute fine-tune on the laptop, using Laya’s RLCD recipe ported to Apple GPUs, lifted Laya from 0.55 to 0.96 AUROC with the hook, at 124 ms per note.
4. Decision models beat the generative baseline on trial prescreening. On TREC trials, the 4B–27B decision models reached AUROC 0.82–0.90, above the generative baseline (0.80). Safe full automation was still limited, so in this setting the models work best as a triage aid.
5. Measure the threshold on your own notes. Thresholds set on synthetic notes overstated safe coverage on hand-written notes by up to 2.6×. For the fine-tuned model, coverage was 100% on template notes but only 38% on hand-written ones.
The recipe
The repo turns the findings into seven steps you can apply to your own checklist:
- One yes/no question per criterion. Spell out thresholds, units and inclusivity.
- Pick a model. Clef-flash 9B and H2O-Lightning-4B were the strongest zero-shot.
- Give the model the arithmetic with a facts hook.
- Combine the answers into a conjunctive score.
- Label 50–200 of your own notes and measure the automation threshold.
- Fine-tune if coverage is too low (about 90 minutes on an M1 Max).
- Deploy with PHI redaction, audit hooks, and clinician review of a sample of automated negatives.
git clone https://github.com/menonpg/clinical-system1 && cd clinical-system1
./setup.sh
app/run.sh # local screening app at http://127.0.0.1:8765. No note leaves the machine.
Why it matters
Hosted LLMs make eligibility screening a data-governance problem. Here the whole pipeline runs on one machine, each answer comes with a calibrated probability, every rejection names its criterion, and the error budget is something you measure on your own notes. Open decision models can screen checklists locally, provided the arithmetic is made explicit and the threshold is measured on the practitioner’s own notes.
- Paper: Checklists at the Speed of Prefill: Local, Calibrated Clinical Eligibility Screening with Open Decision Models. Prahlad G. Menon. arXiv link coming soon.
- Code: github.com/menonpg/clinical-system1 (Apache-2.0)
Research prototype, evaluated on synthetic and public benchmark text. Not a medical device.