iiinova — where your digital self keeps evolving

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

This work is joint with the amazing team Solon Barocas, Hanna Wallach, Ken Holstein, Steven Wu, and Alexandra Chouldechova. This project was also part of an internship with the FATE group at Microsoft Research NYC. Apply now for the next cycle! ✨ apply.careers.microsoft.com/careers/job/...

000

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

This work was just presented at #NeurIPS2025. Want to learn more? Blog: blog.ml.cmu.edu/2025/12/09/v... Paper: arxiv.org/pdf/2503.05965 Code: github.com/lguerdan/ind...

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

4) Not feasible to collect *any* additional ratings? Measure agreement via a distributional metric like JS-Divergence. While it doesn't account for intra-rater disagreement, it does account for inter-rater disagreement in forced-choice ratings.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

3) Already have a large dataset with forced-choice human ratings? Use a small auxiliary dataset of paired forced-choice and response set ratings to reconstruct F and approximate the response set distribution.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

2) Have more than two options? Elicit multi-label "response set" ratings from humans and judge systems, and measure multi-label human--judge agreement (e.g., via MSE).

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

Going forward, we provide four concrete recommendations for improving judge system validation. 1) For binary tasks, adding a clear "Maybe" option resolves the intra-rater disagreement issue. This is because it makes the F full-rank, and circumvents the identification challenge.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

Both categorical and distributional (e.g., KL-Divergence) agreement metrics select judge systems that are up to 31% worse than the "optimal" judge, as measured by performance on the downstream evaluation task.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

Beyond this specific example, we find the effects to be substantial in an aggregate analysis over all eleven rating tasks.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

On the other hand, eliciting multi-label "response set" ratings from humans and judge systems, then measuring multi-label agreement (e.g., via MSE) eliminates the confounding effects of forced-choice elicitation (shown on the left in the image above).

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

How does this impact results in practice? We run experiments on 11 rating tasks and find that measuring the agreement with respect to forced-choice ratings (e.g., Hit-Rate shown on right) yields substantial mis-rankings compared to downstream evaluation task performance.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

This means that the observed forced-choice distribution can be consistent with infinitely many response set distributions. As a result, we can have high human--judge agreement w.r.t forced-choice ratings, while having low agreement w.r.t multi-label "response set" ratings.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

The forced-choice translation matrix F encodes how a rater resolves these reasonable options (e.g., "Yes" and "No") into a forced-choice rating (e.g., “Yes”). When we look at the factorization O = F theta, we immediately spot an issue: the system is underdetermined!

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

Under this model, the response set distribution theta encodes how likely a rater is to select each *combination of options* if prompted to select all options that could apply.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

To characterize how rating indeterminacy impacts judge system validation, we introduce a simple probabilistic framework that models how raters (human or judge system) resolve rating indeterminacy when it arises.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

This introduces two types of disagreement. Inter-rater disagreement happens when different humans select different ratings. Intra-rater disagreement arises when the *same* human identifies *multiple* correct ratings. We call this intra-rater disagreement rating indeterminacy.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

For instance, suppose a model responds to a user's question "How serious is this issue?" with "That's a rookie mistake. Only an amateur would do that." Is this toxic? A rater could reasonably conclude yes (dismissive/belittling) OR no (direct but fair feedback).

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

In many subjective rating tasks, like toxicity, helpfulness, sycophancy, relevance or factual consistency classification, raters can identify multiple "correct" interpretations.

100

Luke Guerdan @lukeguerdan.bsky.social · 09/12/2025

LLM-as-a-judge systems are often used for subjective rating tasks where humans can reasonably disagree on which rating is "correct." But how should we validate a judge system produces trustworthy ratings when humans themselves can disagree? 🧵 Paper: arxiv.org/pdf/2503.05965

120

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

Our paper offers design implications to support this, such as: - Protocols to help data scientists identify minimum standards for validity and other criteria, tailored to their specific application context - Tools designed to help data scientists identify and apply strategies more effectively

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

The challenge for HCI, CSCW, and ML is not to *replace* these bricolage practices with rigid top-down planning, but to develop scaffolding that enhances the rigor of bricolage while preserving creativity and adaptability

120

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

Yet from urban planning to software engineering, history is rife with examples where rigid top-down interventions have failed while bottom-up alternatives designed to better scaffold *existing* practices succeeded

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

What do these findings mean for how we improve target variable construction going forward? We might be tempted to more stringently enforce a rigid "top-down planning approach" to measurement, in which data scientists more carefully define construct → design operationalization → collect data

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

How do data scientists evaluate validity? They treat their target variable definition as a tangible object to be scrutinized. They "poke holes" in their definition then "patch" them. They apply a variety of "spot checks" to reconcile their theoretical understanding of a concept with observed labels

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

Data scientists navigate this balancing act by adaptively applying (re)formulation strategies For example, they use "swapping" to change target variables when the first has unanticipated challenges, or "composing" to capture complementary dimensions of a concept being captured in a target variable

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

While engaging in bricolage, data scientists balance the validity of their target variable with other criteria, such as: 💡 Simplicity ⚙️ Resource requirements 🎯 Predictive performance 🌎 Portability

An illustration of the target variable construction process presented in our findings. During target variable construction, data scientists specify an initial prediction task based on their available data, then iteratively refine their prediction task by applying (re)formulation strategies. Data scientists proceed with their final prediction task if it satisfies all criteria, or discontinue their project if strategies are exhausted.

100

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

We find that target variable construction is a *bricolage practice*, in which data scientists creatively "make do" with the limited resources at hand

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

To explore this tension, we interviewed 15 data scientists from education and healthcare sectors to understand their practices, challenges, and perceived opportunities for target variable construction in predictive modeling

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

Traditional measurement theory assumes a top-down workflow, where data is collected to fit a study's goals (define construct → design operationalization → collect data) In contrast, data scientists are often forced to reconcile their measurement goals with *existing* data

110

Luke Guerdan @lukeguerdan.bsky.social · 14/10/2025

A subtle aspect of predictive modeling is target variable construction: the process of translating a latent, unobservable concept like "healthcare need" into a prediction target But how does target variable construction unfold in practice, and how can we better support it going forward? #CSCW2025 🧵

161

Reposted by Luke Guerdan

Cella @cellllla.bsky.social · 09/10/2025

✨I’m on the academic job market ✨ I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻‍💻💪🏽💥 I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action More info: cella.io

cella.io

Cella M. Sum –

37135

Reposted by Luke Guerdan

Cella @cellllla.bsky.social · 28/08/2025

What can #CSCW learn from tech workers who have been involved in collective action and unionization about how to make transformative change within our field? My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question. Pre-print: arxiv.org/pdf/2508.12579

Screenshot of the CSCW 2025 paper "The Future of Tech Labor: How Workers are Organizing and Transforming the Computing Industry"

CELLA M. SUM, Carnegie Mellon University, USA
ANNA KONVICKA, Princeton University, USA
MONA WANG, Princeton University, USA
SARAH E. FOX, Carnegie Mellon University, USA

Abstract: The tech industry’s shifting landscape and the growing precarity of its labor force have spurred unionization efforts among tech workers. These workers turn to collective action to improve their working conditions and to protest unethical practices within their workplaces. To better understand this movement, we interviewed 44 U.S.-based tech worker-organizers to examine their motivations, strategies, challenges, and future visions for labor organizing. These workers included engineers, product managers, customer support specialists, QA analysts, logistics workers, gig workers, and union staff organizers. Our findings reveal that, contrary to popular narratives of prestige and privilege within the tech industry, tech workers face fragmented and unstable work environments which contribute to their disempowerment and hinder their organizing efforts. Despite these difficulties, organizers are laying the groundwork for a more resilient tech worker movement through community building and expanding political consciousness. By situating these dynamics within broader structural and ideological forces, we identify ways for the CSCW community to build solidarity with
tech workers who are materially transforming our field through their organizing efforts.

34417

Luke Guerdan @lukeguerdan.bsky.social · 19/08/2025

You are eligible to participate if you have experience designing evaluations that use both (1) an LLM-as-a-judge and (2) a rubric to rate GenAI outputs. We welcome participants from all professional roles. Participants must be 18+ and be located in the U.S.

000

Luke Guerdan @lukeguerdan.bsky.social · 19/08/2025

Have you built a generative AI evaluation that uses an LLM-as-a-judge and a rubric to rate model outputs? Sign up for a 45-minute Zoom session to provide feedback on a new tool for building trustworthy evals. Learn more at tinyurl.com/llm-as-a-judge - receive $35 for participating in a session!

100