iiinova — where your digital self keeps evolving

Mohit Chandra @mohit30.bsky.social · 23/06/2025

Hello Athens! 👋☀️ Excited to be attending #FAccT 2025 and presenting our paper “From Lived Experience to Insight” on 24th June at 10:45 AM (New Stage C) dl.acm.org/doi/10.1145/... Would love to catch up with old friends and make new ones and talk about AI and mental health 😄

040

Mohit Chandra @mohit30.bsky.social · 07/01/2025

Finding #6: We examined the actionability of mitigation advices. Expert responses scored the highest on overall actionability in comparison to all the LLMs. While LLMs provide less practical and relevant advice, their advice is more clear and specific. 10/11

Table 3: Mean actionability alignment scores of harm reduction strategies (last column),computed as average of practicality, relevance, specificity, and clarity scores.

100

Mohit Chandra @mohit30.bsky.social · 07/01/2025

Finding #5: LLMs struggle to provide expert-aligned harm reduction strategies with larger models producing less expert-aligned strategies than smaller ones. The best medical model aligned with experts ~71% (GPT-4o score) of the time. 9/11

Table 2: Alignment of harm reduction strategies of various models with the expert’s response.We report the mean and standard deviation for the AlignScore metric GPT-4o score, with the best (bold) and second-best (underline) performing model in each metric highlighted.

100

Mohit Chandra @mohit30.bsky.social · 07/01/2025

Using the ADRA framework, we evaluate LLM alignment with experts across expressed emotion, readability, harm reduction strategies, & actionable advice. Finding #4: We find that LLMs express similar emotions and tones but provide significantly harder to read responses. 8/11

Mean SMOG Scores (for readability) and 95% Confidence Intervals for Various Models (lower values are better).

100

Mohit Chandra @mohit30.bsky.social · 07/01/2025

Finding #1: Larger models perform better for ADR detection tasks (Claude3 Opus led with an accuracy score of 77.41%), but this trend does not hold for ADR multiclass classification. Additionally, distinguishing ADR types remains a significant challenge for all models. 5/11

Table 1: Performance of different models on Binary Detection and Multiclass Classification tasks under Zero-Shot and 5-Shot scenarios.W e report the accuracy score(Acc.) and weighted F1 score as(F1) with the best and second-best performing model metrics in each scenario highlighted in bold and underline, respectively.

100

Mohit Chandra @mohit30.bsky.social · 07/01/2025

We introduce the Psych-ADR, a benchmark with Reddit posts annotated for ADR presence/type, paired with expert-written responses and the ADRA framework to systematically evaluate long-form generations in detecting ADR expressions and delivering mitigation strategies. 4/11

Figure 1: Overview of work; we present two tasks in this work– ADR detection and multiclass classification (RQ1), and Expert-LLM response alignment (RQ2).

100

Mohit Chandra @mohit30.bsky.social · 07/01/2025

Adverse Drug Reactions are among the leading causes of hospitalizations for mental health issues. With existing limitations, LLMs have the potential to detect ADRs and provide mitigation strategies. But do LLMs align with experts? 🤔 We explore this in our work 👇🏼🧵 shorturl.at/bldCb 1/11

(Left) Screenshot of the paper title and author information. (Right) An image with a Llama and human doctor examining a patient for adverse drug reaction. The Llama doctor suspects that the patient is suffering from ADR, whereas human doctor suspects that it is not an ADR.

100