iiinova — where your digital self keeps evolving

Andrew Gordon @andrewgordon.bsky.social · 08/10/2026

This is a super interesting and unique approach to assessing LLM use in online platforms - highly worth a read! 👏

Andrew Gordon @andrewgordon.bsky.social · 06/10/2026

So many thoughts 😅 There is a tonne of work that goes into the platform side to protect against this @sebastiansauppe.bsky.social (too much for a bsky thread). The key here though is that there are extremely effective detection methods that exist for this type of agent, they just arent tested here

010

Andrew Gordon @andrewgordon.bsky.social · 22/09/2026

An agent completes a survey, this is not ground breaking news, yet its led to another set of 'online testing is dead' posts. Remember capability ≠ deployment. Just because it can be done doesn't mean it is being done Also, mouse tracking catches it easily. Design studies with this in mind.

121

Andrew Gordon @andrewgordon.bsky.social · 22/09/2026

This is so important - we've been warning against relying on the types of checks we would use to catch bad participants as a way to catch agents for a long time. Behavioural signatures are critical and researchers need to get used to integrating these types of measures into their studies

010

Andrew Gordon @andrewgordon.bsky.social · 20/08/2026

Our takeaway is that synthetic respondents are predictions of opinion, not measurements. Useful for directional work, but not a substitute for human participants. #syntheticdata #LLMs #surveyresearch [6/6]

020

Andrew Gordon @andrewgordon.bsky.social · 20/08/2026

Also, a warning - our worst method at reproducing the distribution was among the best at calling the plurality winner, so judging synthetic data on one metric rewards exactly the wrong methods.... worth thinking about when anyone tells you their data is X% accurate (on what?!) [5/6]

120

Andrew Gordon @andrewgordon.bsky.social · 20/08/2026

The bigger fix was in changing the prompting method. Eliciting a probability distribution per persona rather than one forced answer cut error by 35-48% while keeping individual-level data. But no method matched the accuracy of a second human sample. [4/6]

110

Andrew Gordon @andrewgordon.bsky.social · 20/08/2026

What is the paradox? Simulating 996 individuals should beat a single zero-context guess.... but it didn't. It roughly 3x'd the distributional error, with personas collapsing onto stereotyped answers. Interviews helped, but not enough to beat the zero-context baseline. [3/6]

110

Andrew Gordon @andrewgordon.bsky.social · 20/08/2026

We fielded a 21-item political and consumer survey to 996 US participants, each of whom also completed an AI-moderated interview. We then tried to reproduce the human data with LLM respondents at increasing levels of fidelity, zero-context aggregate prediction all the way to "digital twins" [2/6]

110

Andrew Gordon @andrewgordon.bsky.social · 04/08/2026

@cschwem2er.bsky.social lets do it! I'll fire you over an email, would be great to hear about what you're working on

110

Andrew Gordon @andrewgordon.bsky.social · 16/06/2026

We're looking for someone who: 1. Has a quant PhD 2. Is based on the East Coast US 3. Has a strong grounding in research methods and work related to sampling (ideally online) 4. Has a track record of high quality publications and conference presentations Any questions just reach out! #hiring

000

Andrew Gordon @andrewgordon.bsky.social · 16/06/2026

This job on my team at @joinprolific.bsky.social is still open (but not for much longer) job-boards.eu.greenhouse.io/prolific/job... If you are an academic or industry researcher and want to work at the coal-face of online research methods then take a look!

lnkd.in

This link will take you to a page that’s not on LinkedIn

100

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

The danger is not that concerns about AI and survey integrity are raised, but that extraordinary claims about failures of infrastructure are being advanced without evidentiary and disclosure standards, risking a misallocation of resources away from the real challenges of AI disruption [6/6]

010

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

Finally, key studies driving this narrative exhibit some clear methodological gaps (withheld code, undisclosed survey instruments, as well as opaque analysis and a lack of benchmarking) making independent verification of the claims impossible. [5/6]

100

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

Second, bypassing in-survey checks made to catch low-quality humans isn't the same as infiltrating a panel like @joinprolific.bsky.social, @cloudresearch.bsky.social, or @verasight.io. Successfully embedding agents within these panels is a far higher bar than commentary suggests. [4/6]

110

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

First, much of the evidence concerning AI contamination conflates humans using AI to assist and agents being able to complete a survey, with the threat of ecosystem-embedded AI agents. The former are known and documented threats, the latter isn't and therefore requires distinct evidence. [3/6]

110

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

We use this letter to respond to the breathless commentary around AI agents constituting an “existential” threat to online research by making three key arguments (full disclosure, we originally submitted this letter to @pnas.org but received a rejection so are publishing it ourselves) [2/6]

100

Andrew Gordon @andrewgordon.bsky.social · 03/06/2026

Sharing our open-letter to the research community “Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial” by @davmicrot.bsky.social, @soubhikbarari.bsky.social, Sunshine Hillygus, Trent Buskirk, and I. [1/6] osf.io/preprints/so...

osf.io

OSF

143

Andrew Gordon @andrewgordon.bsky.social · 09/05/2026

@chrhuber.bsky.social sorry I completely missed this! Did someone get back to you?

010

Reposted by Andrew Gordon

Justin Sulik @justinsulik.bsky.social · 15/04/2026

We have a new preprint that underscores some key claims here: even if one *can* design an agent that gets through a survey fine, it doesn't follow that such agents are undetectable or common. We find that they are far from common! Preprint link in thread👇

1178

Andrew Gordon @andrewgordon.bsky.social · 14/04/2026

If this sounds like the research career opportunity you've been waiting for, I'd love to hear from you. Apply via the link or DM me 👇 [4/4] US applicants: job-boards.eu.greenhouse.io/prolific/job... UK applicants: job-boards.eu.greenhouse.io/prolific/job...

job-boards.eu.greenhouse.io

Research Scientist

Remote, US, East Coast

000

Andrew Gordon @andrewgordon.bsky.social · 14/04/2026

We're looking for someone with: • A PhD in a relevant quant field • Experience in online research methods — sampling, data quality, synthetic data • A track record of publications or public outputs • US east coast preferred, but open to UK also • Postdoc/industry experience preferred [3/4]

100

Andrew Gordon @andrewgordon.bsky.social · 14/04/2026

The hire will own a dedicated research portfolio, design an agenda, publish in high-impact journals, collaborate with leading researchers, and present at conferences. This is an IC role and comes with real autonomy to research the most interesting questions concerning online research! [2/4]

110

Andrew Gordon @andrewgordon.bsky.social · 14/04/2026

Job ad: I'm hiring for a Research scientist to join my team at @joinprolific.bsky.social If you've ever wondered who's working on the hard questions in online research — data quality, sampling methodology, the effect of AI on how research gets done — this is that job. [1/4]

11410

Andrew Gordon @andrewgordon.bsky.social · 02/04/2026

@kwcollins.bsky.social the costs are composited together at the same level of platform type yes @mrandall.bsky.social correct!

000

Andrew Gordon @andrewgordon.bsky.social · 02/04/2026

honestly, I have no idea still see it mentioned at conferences though

000

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

The takeaway: LLM agents are not infiltrating the platforms researchers actually use. Outside of one platform already notorious for pre-LLM bot problems, they just aren't there. The human data quality problem on the other hand is large, systematic, and fixable through platform choice. [9/9]

0101

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

And the cost finding: At a 90% quality threshold, Direct panels cost $8.26 per quality respondent. Marketplace: $74.43. That is not a typo. "Cheap" platforms are often the most expensive data you can buy. [8/9]

260

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

The bigger story is human data quality. Platform type was a far more consequential predictor than anything agent-related. Direct panels outperformed hybrid, which outperformed marketplace, consistently across nearly all measures. [7/9]

140

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

And the MTurk detections didn't look like LLM agents. Poor writing quality, fast completions, clustered arrivals. Classic bot behavior. When we ran real LLM agents through the same survey, they outperformed average humans and vastly outperformed the flagged responses. [6/9]

150

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

So are AI agents infiltrating surveys? Not really. Meaningful detections were almost exclusively on MTurk (11-16%). Every other platform: at or below 1% for our primary detection flag [5/9]

175

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

Then we recruited 5,200 respondents across 10 platforms spanning direct (Prolific, CloudResearch Connect, Verasight), hybrid (Dynata, Prodege), and marketplace (Cint, Qualtrics, Purespectrum, Prime Panels), and MTurk. Same survey for everyone. 7 behavioural quality measures. Full metadata. [4/9]

130

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

First we validated our detection methods against real agents (Claude, ChatGPT, Gemini, Perplexity, plus a custom white-hat agent) vs real humans. Primary method: 100% sensitivity, 100% specificity. Secondary behavioural battery: 92% sensitivity, 99.2% specificity. [3/9]

140

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

There's been a lot of alarm about LLM agents polluting survey samples. Capability demos are impressive. But capability is not the same as deployment within an ecosystem. We wanted to know what's actually happening in the platforms researchers use. [2/9]

140

Andrew Gordon @andrewgordon.bsky.social · 01/04/2026

New preprint out today (osf.io/preprints/ps...). We tested whether AI agents are actually infiltrating online surveys. Spoiler alert: they aren't Thread 🧵 [1/9]

osf.io

OSF

213162

Reposted by Andrew Gordon

Prolific @joinprolific.bsky.social · 31/03/2026

Starting today, if an AI agent is detected in your Prolific study, we’ll give you twice the cost of that non-human participant back. You pay for human data. You expect human data. We’re backing that with our 100% Human Guarantee. Learn more: www.prolific.com/100-human-gu... #AcademicSky #Research

0133

Andrew Gordon @andrewgordon.bsky.social · 20/03/2026

Fantastic example of researchers working together (and the utility of rebuttals to published work). I think we all agree that this is an area we need to invest time in, but we also need to be very careful that conclusions/interpretations are warranted from the data we collect.

110

Andrew Gordon @andrewgordon.bsky.social · 12/03/2026

@drbarner.bsky.social you might want to read this bsky.app/profile/ache...

010

Andrew Gordon @andrewgordon.bsky.social · 10/03/2026

Your letter very clearly reads as saying that there are bots, and you use that to question the integrity of online sampling... but you agree with the notion that your measures are not sufficiently establishing to make such a claim. Do you not think an amendment to your published letter is required?

110

Reposted by Andrew Gordon

Andrey Chetverikov @achetverikov.bsky.social · 04/03/2026

Recently, van der Stigchel and colleagues posted a provocative commentary suggesting that we should be wary of bots in online behavioral data collection (🧵by @cstrauch.bsky.social here: bsky.app/profile/cstr...). But should we? Here is my response letter osf.io/preprints/ps.... 1/5

65533

Andrew Gordon @andrewgordon.bsky.social · 23/02/2026

On Thursday I'll be taking part in a roundtable “How real is the LLM threat to online research in academia?” for @joinprolific.bsky.social alongside @davmicrot.bsky.social, Michael Nicholas Stagnaro, and Raluca Rilla. Sign up here: lnkd.in/em8-MpjN

lnkd.in

How to Bot-Proof Your Online Research

This series cuts through the noise to give researchers practical, actionable strategies for protecting their work from AI agents and bots

031

Andrew Gordon @andrewgordon.bsky.social · 29/01/2026

Interval strongly predicted retention: 1-week → 80% completed all sessions; 4-week → 50%. Payment had no significant effect. Participants higher in routine showed better retention; those higher in automaticity showed worse Read the paper here: osf.io/preprints/ps...

osf.io

OSF

000

Andrew Gordon @andrewgordon.bsky.social · 29/01/2026

What drives retention in online longitudinal research? We conducted an experiment (N=1,798) on @joinprolific.bsky.social, orthogonally manipulating payment rate (£6–£9/hr + bonus) and session interval (1, 2, or 4 weeks) across five sessions. The findings challenge some common assumptions 👇

172

Andrew Gordon @andrewgordon.bsky.social · 09/01/2026

Hey @rory-stewart.bsky.social @alastaircampbell2.bsky.social @therestpolitics.bsky.social I asked your favourite question to 1,936 US Adults "Who do you believe is the biggest threat to global order and security?" 33.8% of Americans rate the US as the biggest threat.... 🤯

000

Andrew Gordon @andrewgordon.bsky.social · 23/12/2025

@michaeljkane.bsky.social pop me over a message and will do what I can to help!

100

Andrew Gordon @andrewgordon.bsky.social · 19/11/2025

Second, and most importantly, the economics of it don't make sense, even at $0.05/response. For someone to scale this approach would require multiple user accounts (which we have robust guards against), meaning the break-even point for a bad actor is likely impossible to reach

020

Andrew Gordon @andrewgordon.bsky.social · 19/11/2025

First, the barrier to entry here is really high. Creating a bot like this is not trivial, and required an academic team to design and implement. Current 'naive' agents such as that offered by ChatGPT are simple to catch - track mouse moves, typing speed, or even simple reverse shibboleths

110

Andrew Gordon @andrewgordon.bsky.social · 19/11/2025

Lots of chatter about this paper currently. Its a stark warning, but at present I see this as a stark warning of what might come, not what is happening now. As a research community we need to see it as a call-to-arms to develop new strategies, NOT a call to abandon online sampling. Reasoning below

153