iiinova — where your digital self keeps evolving

To read more about mixed subjects designs in casual inference applications, check out this preprint with @austin-van-loon.bsky.social: osf.io/c2rjp/overview Full paper on the IRT-focused method coming soon!

osf.io

OSF

100

If you're curious how and why this works, I've got a more technical vignette that also includes tips on getting better item responses out of your LLM, as well. Check it out here: klintkanopka.com/mixedsubject...

Understanding Ability-Risk Tuning

100

We often don't consider it, but item parameter estimation error propagates into ability estimation error. Guarantees include unbiased parameter estimates and ability estimation error less than or equal to that of items calibrated on human subjects alone.

100

Often LLMs can generate item responses that don't behave like human responses. Using a prediction-powered inference approach, we tune how much to rely on LLM data with the goal of minimizing downstream ability estimation error. If the LLM responses are trash, mixedsubjectsirt tells you to kick rocks

100

If you're trying to include LLM-generated item responses in psychometric work, you should check out my mixedsubjectsirt package that's now available on CRAN! Importantly, this is a statistical and LLM-agnostic approach with strong efficiency guarantees (a thread). klintkanopka.com/mixedsubject...

Item Response Theory Calibration with a Mixed Subjects Design

Integrates large language model generated item responses into psychometric calibration studies through a mixed-subjects design for unidimensional two-parameter and one-parameter logistic item response...

130

Klint Kanopka @klint.bsky.social · 15/01/2026

Excited to share a new call for papers for a special issue in Psychometrika focused on Data Intensive Methods in Psychometrics that I'll be guest editing with @kyliegorney.bsky.social, @jmbh.bsky.social, @leonievogelsmeier.bsky.social, and Ben Domingue: www.psychometricsociety.org/post/call-sp...

psychometricsociety.org

Call for Papers: Psychometrika Special Issue - Psychometric Society

Data Intensive Methods in Psychometrics

0157

Klint Kanopka @klint.bsky.social · 18/11/2025

Tomorrow at the PRIISM seminar: @crahal.com is going to talk to use about potential pitfalls and sources of error caused by psuedorandom number generation in social science research. Talk is at 11a eastern time on Zoom and open to anyone. RSVP here: steinhardt.nyu.edu/events/respo...

PRIISM | Responsible use of Pseudo-Random Number Generators in Applied Scientific Research

Join PRIISM and Dr. Charles Rahal to learn how to use pseudo-random number generators (PRNGs) responsibly in research.

040

Klint Kanopka @klint.bsky.social · 30/10/2025

I've said this before, but I really love my statistical computing course. Tomorrow is one of my favorite days, where we start using MCMC. This semester I've redone all of my slides and am hosting them on a public site, so feel free to check them out: klintkanopka.com/lecture-slid...

Klint Kanopka @klint.bsky.social · 18/09/2025

Slidev Workspace

1190

My co-author demanded that the icon actually be cute instead of the horrific thing I presented at IMPS.

060

Klint Kanopka @klint.bsky.social · 18/09/2025

Wanted to hype up a software package I'm debuting today. If you do computer adaptive testing (CAT) research and are developing your own item selection/parameter update/exposure control algorithms, take a look at meow - a unified framework to make CAT simulation easier! klintkanopka.com/meow/

Unified Framework for Computer Adaptive Testing Simulations

Provides an extensible framework for conducting simulations to compare data generating processes, item selection algorithms, parameter update algorithms, and stopping rules in computer adaptive testin...

240

Klint Kanopka @klint.bsky.social · 18/08/2025

Same!! Gut feel is that it’s getting attributed to measurement error because they’re not really set up to disentangle the two

010

Klint Kanopka @klint.bsky.social · 18/08/2025

“When achievement follows the state-space model and there is no heterogeneity in growth rates”

100

Klint Kanopka @klint.bsky.social · 06/08/2025

Apparently some people like listening to me blather on about things! Thanks to @crahal.com for inviting me, this seminar has some amazing speakers lined up and brilliant attendees.

020

Klint Kanopka @klint.bsky.social · 20/07/2025

Incredibly proud of the work @ruitings.bsky.social did, and it’s even more impressive that she just finished the first year of her master’s program!

050

Reposted by Klint Kanopka

Ruiting Shen @ruitings.bsky.social · 19/07/2025

I gave my first conference paper presentation at IMPS 2025 on process data analysis (credits to my amazing advisor @klint.bsky.social for all his support and advice) I also did my first data challenge and got a third prize (credits to my wonderful teammates Jessie and Na) #IMPS2025 #Psychometrics

041

Reposted by Klint Kanopka

Charlie Rahal @crahal.com · 17/07/2025

@ramina-sotoudeh.bsky.social gave an AMAZING talk yesterday as part of the #metrics-and-models seminar series👏. Next (30.07), we have @klint.bsky.social on "Reproducing Expert Judgement with Shortened Surveys using Simulated Annealing"🔥! See website for more; metrics-and-models.github.io!

021

They aren’t, a common convention in our field is to use thetas for latent qualities of persons

100

Amazing. I used to have \hat hat I got from mathpix. For some reason the closest thing I have to photographic evidence makes me look like I’m about to “vote by mail.” I saw Russ Almond wearing a fishing vest at NCME in Toronto and became obsessed with the vibe, but I don’t carry enough snacks.

000

We are estimating latent traits here

110

Charlie Rahal @crahal.com · 10/06/2025

Shoutouts to @pmetricsoc.bsky.social and University of Minnesota Ed Psych for having the most unhinged (free) conference merch I’ve ever seen

1221

Reposted by Klint Kanopka

🔥Coming July 2: 'Metrics and Models'!🔥 A new seminar series on advanced modelling w/ impact across health/society. 🌍 Open to *ALL*! 🕑 Weds 2pm UK alternating weeks (online) 🔗 metrics-and-models.github.io 📬 metrics_and_models-subscribe@maillist.ox.ac.uk Brilliant speakers. *Please share*!

12617

Klint Kanopka @klint.bsky.social · 28/04/2025

Join us for our last (hybrid) PRIISM seminar of the academic year on 5/7! @austin-van-loon.bsky.social will present an experiment looking at how generative AI in admission essays may reshape the demographics of admitted students and how merit is evaluated. RSVP: steinhardt.nyu.edu/events/compe...

Competitive Self-Presentation and GAI | PRIISM

Join PRIISM and Dr. Austin van Loon to learn about an ongoing online experiment investigating how generative AI in college admission essays may reshape the demographic makeup of admitted students and ...

051

Klint Kanopka @klint.bsky.social · 28/04/2025

That's a handsome garment, Andrew. Where might someone be able to obtain one? @ncme38.bsky.social

100

Reposted by Klint Kanopka

Tracy Sweet @tracysweet.bsky.social · 28/04/2025

Anyone interested in learning social network analysis in R? I'll be teaching a workshop soon. May 15/16th. umd-cilvr.catalog.instructure.com/courses/intr...

Two-day online short course: Introduction to Social Network Analysis taught by Tracy Sweet. $395 and $195 student discount!

084

Klint Kanopka @klint.bsky.social · 01/04/2025

Not quite- you’re way closer than you think! What you described is referred to as “differential item functioning.” This talk is about something akin to a specific and narrow version of that.

110

Klint Kanopka @klint.bsky.social · 01/04/2025

Not quite - @jbgilbert.bsky.social can chime in, but the application is in RCTs. When the goal is to measure the effect of an intervention or treatment condition, it may be that only a subset of responses to items show a treatment effect. Maybe a math test only shows an effect on geometry items, say

100

Klint Kanopka @klint.bsky.social · 01/04/2025

Join us at the intersection of causal inference and psychometrics as we host Josh Gilbert (@jbgilbert.bsky.social) at the PRIISM Seminar. Wednesday @ 10a: Estimating heterogeneous treatment effects with item-level data: Insights from Item Response Theory steinhardt.nyu.edu/events/estim...

PRIISM | Estimating heterogeneous treatment effects with item-level data

Join PRIISM and Josh Gilbert to learn how to unmask hidden treatment effects within individual test items using Item Response Theory and to understand how to overcome the limitations of traditional si...

282

Klint Kanopka @klint.bsky.social · 28/03/2025

Incredibly excited to have this finally come out! Model evaluation should be about comparisons, so we have a metric that puts comparisons in predictive performance on a common scale. I can’t make a thread about this better than @crahal.com, so I’ll let him take it away.

042

Manhattan of all places. I used to obsessively document them when I lived in Philly. In 2018, there were three between Bryant Park and Radio City, and this is the only still left.

070

Saw something nice on a walk today

2211

Open: The data are licensed for reformatting and public reuse. Their provenance is documented and code used to transform original data to meet the IRW data standard is available.Harmonized: The data a...

The elephant in the room, of course, is that in educational measurement, we use data collected from kids. The protections they need makes the logistics trickier. The Item Response Warehouse is one step toward democratizing psychometric research, but more must be done. datapages.github.io/irw/ 2/2

datapages.github.io

Item Response Warehouse

171

Andrew Ho @andrewdeanho.bsky.social · 14/03/2025

I've said this before, but the single greatest force holding back psychometric research is data availability. Researching innovative assessments requires response data, but those data are owned by those who own the assessments. This gives them carte blanche to gatekeep who can do this work. 1/2

1180

Reposted by Klint Kanopka

Research with restricted-use data is more precise, powerful, and relevant. Faculty and staff will be working through the weekend to answer important research questions before we no longer can. This is not normal, necessary, or efficient.

36038

I’ll add that I have a wonderful master’s student who has spent this entire academic year working to prepare to do interesting work with this data that was going to form the basis for her PhD applications. We’ll pivot and persevere as best we can, but she’s crushed right now.

1142

Klint Kanopka @klint.bsky.social · 03/03/2025

The loss of the NAEP restricted-use process data is crushing. It was the most reasonable to acquire dataset that is useful for actual measurement work (as opposed to nonsense mining). The data hoarding practice of private ed and testing companies harms research and students. They must fill the gap.

2147

Can AI-powered textbots improve survey interviews? Join PRIISM and @soubhikbarari.bsky.social to examine how they can enhance response quality and scalability, while considering challenges in bias and respondent experience. If you're in NYC, join us in person! steinhardt.nyu.edu/events/ai-as...