iiinova — where your digital self keeps evolving

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Overall, we reveal a trade-off between the computational efficiency of specialized EHR models and the portability of LLM-based embeddings. For long-context EHR data, LLM embeddings are a stronger option than encoder-only models such as (Modern)BERT. 13/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Questions & limits: - Compute and memory costs of LLMs - A count model outperformed both (0.777 vs. 0.769 AUROC) - The EHR serialization contains only one occurrence per code and no time - CLMBR vs. Qwen3 differ in scale, architecture and pretraining data 12/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

We also tested a Qwen3-8B decoder scoring Yes/No tokens, plus LoRA fine-tuning of the decoder and the embedding model. With limited hyperparameter tuning due to computational constraints, neither surpassed the frozen Qwen3-Emb-8B + simple classifier baseline. 11/14

101

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

In our ablation studies we found that only Qwen3-Emb-8B handles the full 8192-token context robustly. Qwen2-Emb-7B and Llama3.1-LLM2Vec-8B degrade at longer inputs and time windows. Model scale alone does not ensure strong long-context EHR representations. 10/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

In the external validation on the UKB, Qwen3-Emb-8B consistently performed slightly better than CLMBR-T-Base. Only 16% of UKB codes map to CLMBR's vocabulary, but restricting Qwen3 to them still leaves a small benefit, hinting at better generalization (Figure S13). 9/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Newly added few-shot results for harrier-oss-v1-27b [4]: it further improves over Qwen3-Emb-8B across all four task groups, while CLMBR-T-Base remains superior for operational outcomes from 32 training examples on. [4] huggingface.co/microsoft/ha... 8/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

In the few-shot setting, Qwen3-Emb-8B stays on par with CLMBR-T-Base but clearly beats the count baseline at small shot sizes. It also consistently outperforms the commonly used BioClinicalBERT on chunked 512-token inputs. 7/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

On EHRSHOT, Qwen3-Emb-8B performs on par with CLMBR-T-Base for all training examples (0.769 macro-AUROC each), but at higher compute cost: 6 min vs. ~22 h to encode. Post-acceptance, we also tested Microsoft's new harrier-oss-v1-27b that alone reached 0.788 (CIs overlap). 6/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

We initially tried a more complex EHR serialization aggregating labs and vitals, but recent models like Qwen3 seem less dependent on it. See also [3], which uses an LLM-generated rubric to restructure the EHR before embedding. @ilkerdemirel_ [3] arxiv.org/abs/2603.11679 5/14

arxiv.org

LLMs can construct powerful representations and streamline sample-efficient supervised learning

As real-world datasets become more complex and heterogeneous, supervised learning is often bottlenecked by input representation design. Modeling multimodal data, such as time-series, free text, and st...

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Our EHR serialization uses minimal preprocessing: a newline-separated list of code descriptions using the most recent occurrence of each, with optional units and values. Recent LLM embedding models are instruction-tuned, so we prepend a short task-specific instruction. 4/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Text embeddings were commonly computed with BERT-style encoders. Recently, embedding benchmarks such as MTEB [2] have driven a shift toward decoder LLMs converted into embedding models, adding instruction tuning and much larger context sizes. [2] huggingface.co/spaces/mteb/... 3/14

huggingface.co

MTEB Leaderboard - a Hugging Face Space by mteb

Embedding Leaderboard

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

EHR FMs are pretrained on medical codes (a). LLMs are pretrained on text, then turned into embedding models via contrastive learning (b) [1]. We use both approaches to encode EHRs and train a simple classifier on 15 EHRSHOT + 25 UKB tasks (c). [1] arxiv.org/abs/2404.05961 2/14

100

Stefan Hegselmann @stefanhgm.bsky.social · 28/07/2026

Can general-purpose LLMs encode raw EHR data for clinical prediction? We tested this on EHRSHOT and the UK Biobank: LLM embedding models perform on par with an EHR FM (CLMBR-T-Base) without clinical pretraining or a fixed vocabulary. Paper: www.nature.com/articles/s41... 1/14

nature.com

Large language models are powerful electronic health record encoders - npj Digital Medicine

npj Digital Medicine - Large language models are powerful electronic health record encoders

121

Stefan Hegselmann @stefanhgm.bsky.social · 01/12/2025

At #ML4H2025 and #NeurIPS2025 in San Diego this year! Interested in EHR foundation models and using LLMs for clinical NLP — but always happy to chat about anything ML-for-healthcare. If you want to connect, hit me up / DM me!

020

Stefan Hegselmann @stefanhgm.bsky.social · 25/11/2024

Would be great to be added as well :)

000