iiinova — where your digital self keeps evolving

Daniel MacArthur

@dgmacarthur.bsky.social

5.7K followers 1.3K following 70 posts

Genomics, big data, open science, diversity. Director of the Centre for Population Genomics, focused on building a more equitable future for genomic medicine. Opinions my own.

Posts Replies Media

Daniel MacArthur @dgmacarthur.bsky.social · 05/04/2025

Reckon we still have a while before ChatGPT replaces genetic counselors

ChatGPT chat interface. The prompt is “draw a pedigree illustrating second cousins” and the response is a nonsensical pedigree including sibling consanguinity between a pair of brothers

611220

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

Of course EvAgg isn't intended to be used as a standalone tool, but as a support for variant scientists - so the key question is whether it helps them. And yes, it does, in a whole variety of ways; users also *really* liked using it, although they were (understandably) still wary of its accuracy.

User study results: EvAgg use reduced the time to complete a case by 34%, and resulted in a >2x increase in the number of papers assessed by the curator.

Survey of users shows very strong favourability for user experience, but mixed trust ratings - although with over 50% of users indicating a positive trust rating.

130

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

Overall, EvAgg was *superb* at finding relevant papers in the provided set, with a recall of 0.97, but with a precision of 0.83. It also does a pretty good job of extracting relevant content, though there's clearly plenty of further room for improvement here.

Benchmarking overview and results. (A) Individual steps of the EvAgg pipeline were separately evaluated against the curated dataset to assess tool performance in different aspects of the task. (B) Task performance when benchmarked against the evaluation subset from the curated dataset: precision and recall for the paper selection and observation finding tasks, accuracy (i.e., the frequency of matching curated dataset content) for the content extraction task.

120

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

It's worth noting that benchmarking against human performance for this kind of complex task is... non-trivial. Interestingly, later assessment of points where EvAgg and the human-curated truth set disagreed suggested that the AI was probably more correct about half the time.

130

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

That's cool in theory, but does it actually work? To test that, the team chose 45 disease-associated genes and manually curated a bunch of literature on those genes. Kudos to the variant science team, including Lynn Pais, Sam Bryen, Emily O'Heir, and Chrissy Austin-Tse.

Benchmark curated dataset generation process. 1) Forty-five genes were
randomly selected from the subset of ClinGen submissions to GenCC with only moderate, limited, or no known gene-disease relationship (GDR). 2) Approximately 4400 papers were returned by a PubMed search for these genes and manually assessed for diagnostic relevance using a standardized rubric, with ~400 papers deemed relevant (Supplementary methods). 3) From these ~400 relevant papers, we selected up to 2 papers for each of the 45 genes and curated detailed content for observations of individuals with genetic variants in the gene of interest. The curated dataset, consisting of the complete list of ~400 relevant papers and curated content for ~200 observations, was used for subsequent benchmarking.

120

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

From a variant curator's perspective, the key thing is that EvAgg can take a gene symbol as input, extract a bunch of pertinent information from multiple papers, and then deliver a handy table summarising that information.

Figure showing the type of table generated by EvAgg, which starts with a gene symbol and produces structured information on individual papers analysed, the individual patient IDs, variant HGVS nomenclature, associated phenotypes, population frequency, variant type, zygosity, and inheritance modes.

130

Daniel MacArthur @dgmacarthur.bsky.social · 15/03/2025

Our amazing Microsoft colleagues, including Hope Twede and Ashley Conard, developed the Evidence Aggregator (EvAgg), which leverages OpenAI tools including GPT-4o to select relevant papers, extract useful information, and then provide an accessible summary about a query variant/gene.

Figure 1 from the EvAgg paper, which compares the traditional manual workflow of variant curators to the ability of the EvAgg tool to assess multiple papers, extract information, and provide a structured summary.

A more detailed overview of the EvAgg tool, including modules for paper selection, observation finding, and content extraction, and the provision of a structured summary output.

120

Daniel MacArthur @dgmacarthur.bsky.social · 19/12/2024

Excited to be tutoring at the Leena Peltonen School of Human Genetics on July 27-31, alongside a stellar crew. If you’re a late-stage or recently graduated PhD student this is an awesome opportunity to get 1:1 time with faculty at the cutting edge of genomics. Apply by March 7th at lpshg.com

Flyer for the LPSHG - details in the post

0103

Daniel MacArthur @dgmacarthur.bsky.social · 04/10/2024

GPT-4 gets it wrong, but o1-preview has no trouble.

010